1. Field of the Invention
The present invention relates generally to the field of image processing for correction of image distortion produced by a wide-angle imaging system. In particular, the invention relates to a system permitting interactive navigation within a wide-angle image such as a panoramic image in a manner that is free of image distortion.
2. Description of the Prior Art
Recently, there has been unprecedented interest in research and application development in the fields of immersive panoramic imaging, telepresence and virtual reality. Immersive panoramic imaging applications permit a user to immerse himself in a representation of a remote scene with the ability to look in any desired direction in the scene, limited only by the field of view of the imaging system employed. Immersive telepresence applications permit live omni-directional viewing of remote locations while immersive virtual reality generally involves the synthesis of immersive content.
A major problem with immersive panoramic imaging systems is how to encode enough visual information to make the system practical and cost-effective. One solution to this problem involves using an orientation sensing device to sense the orientation of the user and then using the input to rotate a conventional camera to cover the field of view indicated by the user""s orientation. Such a system is described in U.S. Pat. No. 4,728,839 awarded to Coughlan et al. Apart from the obvious handicap of not being able to permit more than one user to comfortably navigate the remote scene, the inherent delay between the transmission of the orientation signal and the positioning of the camera to coincide with the desired orientation severely limits the practical value of this system. A more elegant solution to the problem is to use a panoramic imaging system. U.S. Pat. No. 4,566,763 awarded to Greguss Pal describes one such solution. The fish-eye lens system described in U.S. Pat. No. 5,434,713 awarded to Sato Haruo is yet another solution. Panoramic imaging systems, however, introduce significant amounts of non-linear geometric distortion into the images they produce and so for these images to be viewed comfortably by a human being, they need to be de-warped. The distortion is generally referred to as perspective distortion in the literature and the process of correcting the distortion is referred to as the generation of perspective-corrected views.
Several solutions to the perspective distortion problem exist. One such solution is described in U.S. Pat. No. 5,185,667 awarded to Steven Zimmermann. Zimmermann describes a system utilizing complicated trigonometric transformations and assuming a mathematically ideal fish-eye lens. Since real fish-eye lenses deviate somewhat from the ideal, the application of the system described in the ""667 patent is at best limited to situations where the inevitable errors arising from the assumption of idealness can be tolerated. Even in such applications, the method cannot be extended to panoramic imaging systems other than the fish-eye lens. The solution described in U.S. Pat. No. 6,005,611 awarded to Gullischen et al remedies some of the limitations of the system presented in the ""667 patent. The 611 patent uses control points obtained via a set of projection equations and a global bivariate polynomial to correct the perspective distortion. Although the solution presented in the ""611 patent is simpler and faster than that of the ""667 patent, it relies on the use of the perspective projection equations of the fish-eye lens for the generation of the control points. Since these equations assume an ideal fish-eye lens, the method is of limited practical importance. Furthermore, attempting to extend the method of the ""611 patent to other panoramic imaging systems would require the determination of the projection equations of those panoramic imaging systems. The determination of these equations, where feasible in the first place, is at best tedious. Usually, the projection equations assume mathematical idealness for the panoramic imaging systems and as such the method of the ""611 patent still suffers one of the major drawbacks of the method described in the ""667 patent.
In view of the foregoing, there has been a substantial need for a system that permits the construction of perspective-corrected views from an arbitrary panoramic imaging system that obviates the need to derive the projection equations of the panoramic imaging system or to assume a mathematically ideal panoramic imaging system. There has also been a need for a system that can permit such construction at real-time (typically 30 frames per second) rates on a general-purpose computer.
Furthermore, there has been a substantial need for a system that permits the construction of perspective-corrected views from an arbitrary panoramic imaging system to an arbitrary degree of accuracy so that systems relying on accurate measurements based on the perspective-corrected views could be built.
It should be noted that although in the following the features of the present invention will be described and claimed with respect to a panoramic navigation system, it will become apparent that the invention teaches a method and apparatus for training and utilizing a neural network which would be applicable to obtaining undistorted images from other distorted wide-angle images, so that the scope of the invention is not limited to processing of panoramic images.
In the following, the term xe2x80x9cimage distortionxe2x80x9d is to be understood as applying to both inherent (predictable) distortion, such as occurs in an annular image of a panoramic 3-dimensional scene, and to complex non-linear distortion such as optical distortion produced by an imaging system, whereby directions of incident light rays which pass through a reference origin point of an optical lens unit are not accurately mapped onto image positions.
It is an objective of the present invention to overcome the disadvantages of the prior art set out above, and in particular to provide an image navigation system whereby an arbitrary region of a distorted panoramic image which has been captured by an imaging apparatus can be selected to derive a corresponding view image that is substantially free from distortion.
To achieve that objective, the invention permits the construction of a panoramic navigation system, in which image distortion correction is accomplished by a neural network. The neural network is trained beforehand by utilizing a calibration pattern having an array of visible calibration points defined thereon. Specifically, the calibration pattern is disposed surrounding a lens unit of a panoramic imaging system, and a (distorted) calibration panoramic image of the calibration pattern is captured by that system. The respective view directions of the calibration points on the calibration pattern with respect to a reference origin point of the lens unit are calculated or measured, and these directions are then matched against the respective positions at which the corresponding calibration points appear within the calibration image, e.g., with each of the directions specified as a pair of (elevation, azimuth) angular coordinates. The neural network is then trained, by using the set of calibration directions as inputs to the neural network, with the corresponding response outputs from the network being compared with sets of coordinates expressing the corresponding positions of the calibration points within the calibration image, to obtain error quantities for use in modifying internal parameters (e.g., weight values of neuron connections) of the neural network, using a suitable algorithm, until it is judged that a sufficient degree of training has been achieved.
Once the neural network has been trained, then when a portion of an arbitrary distorted wide-angle image is to be displayed as an undistorted sub-image, e.g., displayed in a view window formed of a rectangular array of pixels in a display screen, a specific viewing direction is specified for the sub-image by a user, and is thereby assigned to a centrally located pixel of the view window. Next, for each of the other pixels of the view window, a corresponding view direction is calculated based upon the view direction of the central pixel and the position of that other pixel with respect to the central pixel.
The set of view directions thus obtained are then successively inputted to the trained neural network to thereby obtain, as responses from the neural network, information specifying positions within the distorted wide-angle image that respectively correspond to the inputted directions.
The video attributes of pixels at each of these positions in the distorted wide-angle image (i.e., with each such position being matched to a specific view direction and hence to a specific pixel position in the view window) are then extracted, and assigned to the corresponding pixels of the view window. In that way, an undistorted sub-image appears in the view window, which corresponds to an image which would be seen by looking at a selected part of the original real-world scene along the specified view direction.
The generation of the location data by the neural network primarily involves a simple series (the number depends on the complexity of the distortion of the panoramic imaging system) of multiplications and additions and for most available panoramic imaging systems the number of multiplications and additions required is low enough to permit simultaneous real-time navigation of the panoramic image by a plurality of users on a general-purpose personal computer. The undistorted sub-image, formed of the data extracted from the distorted panoramic image as described above, is then displayed on a suitable display system such as a computer monitor.
In the present invention, the formation of a distorted 2D image from the undistorted 3D real world scene by a panoramic imaging system is considered as an input-output mapping of directions defined with respect to a specific point in three-dimensional space onto respective points in a two-dimensional image. Artificial neural networks have been shown to be capable of approximating an arbitrary input-output mapping to an arbitrary degree of accuracy. The ability of a neural network to approximate the characteristic distortion of the panoramic imaging system to an arbitrary degree of accuracy permits the use of this invention in applications (such as machine vision-based industrial process control) that rely on accurate measurements of parameters from panoramic images.
The invention relies only on sample data from an image captured using the particular panoramic imaging system in use and so obviates the need to make any assumptions of mathematical idealness about the panoramic imaging system. Furthermore, the derivation of the projection equations of the panoramic imaging system is rendered unnecessary. Hence, any panoramic imaging system can be used and its characteristic distortion can be approximated to any desired degree of accuracy.
Accordingly, it is an object of the present invention to provide a system for the construction of arbitrary distortion-corrected views from a panoramic image captured with an arbitrary panoramic imaging system.
It is another object of the present invention to permit a plurality of users to simultaneously view the same or different perspective-corrected views of a panoramic image captured with an arbitrary panoramic imaging system.
It is yet another object of the present invention to permit the approximation of the characteristic distortion of an arbitrary panoramic imaging system to an arbitrary degree of accuracy so that systems relying on accurate measurements based on the perspective-corrected views could be built.
Additionally, it is an object of the present invention to achieve all the foregoing without requiring the determination of the projection equations of the panoramic imaging system or any assumptions of idealness concerning the panoramic imaging system.
These and further objects of the present invention will become clear in the detailed description that follows.